DeepSynth: Synthesizing A Musical Instrument With Video

This paper introduces DeepSynth, an end-to-end neural network model for generating the sound of a musical instrument based on a silent video of it being played. We specifically focus on building a synthesizer for the piano, but the ideas proposed in this paper are applicable to a wide range of musical instruments. At a high level, the model consists of a convolutional neural network (CNN) to extract features from the raw video frames and two stacked neural autoregressive models, one to encode the spatiotemporal features of the video and the other to generate the raw audio waveform.